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Nations  nearly  always  go  into  an  armed  contest  with  the  equipment  and  methods  of  a 
former  war.  Victory  always  comes  to  that  country  which  has  made  a  proper  estimate 
of  the  equipment  and  methods  that  can  be  used  in  modern  ways. 

—Maj  Gen  Billy  Mitchell 


Equipping,  including  research  and  de¬ 
velopment,  is  a  primary  responsibility 
of  the  Air  Force.1  Yet,  a  loss  of  exper¬ 
tise  during  acquisition-reform  initiatives 
and  a  lack  of  immediate  and  continuous 
involvement  of  test  professionals  have 
caused  the  service  to  struggle  in  its  at¬ 
tempts  to  execute  this  critical  task  properly. 
Within  the  defense  acquisition  corps,  these 
individuals  contribute  critical  capabilities 
and  expertise  to  the  mission  of  supporting 
the  materiel  needs  of  the  war  fighter.  To  be 
fully  effective,  they  must  become  involved 
in  this  acquisition  process  at  the  earliest 
stages.  A  proposed  cadre  of  test  professionals 
strikes  a  balance  between  system/mission 
experts  and  developmental  test  experts. 
These  groups  are  developed  along  separate 
career  paths  that  provide  both  recent  opera¬ 
tional  experience  and  profound  technical 
expertise  to  decision  makers  in  the  acquisi¬ 
tion  arena.  A  cadre  of  deliberately  devel¬ 
oped  test  professionals  also  seeds  the  ranks 
of  senior  officers  with  direct  experience  in 
acquisition.  The  result  is  a  full  integration 
of  such  professionals  across  a  system's  life 
cycle,  from  initial  definition  of  require¬ 
ments  through  development  and  initial  op¬ 


erating  capability  to  sustainment  of  war¬ 
fighting  capability  in  our  nation's  defense. 

A  Brief  Sketch  of 
Air  Force  Acquisition 

Report  after  report  has  shown  that  there 
are  fundamental  problems  with  the  way 
we  buy  major  weapons  systems. 

—Senator  Carl  M.  Levin,  6  May  2009 

The  relationship  between  the  govern¬ 
ment's  and  industry’s  conduct  of  flight  test 
has  always  provided  a  constructive  tension 
designed  to  serve  the  requirements  of  the 
war  fighter  while  pushing  the  leading  edge 
of  existing  technology.  Industry  offers  inno¬ 
vative,  quality  solutions  to  the  war  fighter’s 
requirements  while  government  testers  en¬ 
sure  that  the  products  meet  those  require¬ 
ments.  The  military  has  recognized  the 
need  to  develop  its  own  standards  and  per¬ 
form  an  independent  evaluation  of  com¬ 
mercially  produced  aircraft  since  their  ini¬ 
tial  use  in  World  War  I.  The  Air  Corps  Act  of 
1926,  however,  reduced  military  flight  test 
and  evaluation  to  brief  acceptance-test  pro¬ 
grams.  By  the  end  of  World  War  II,  so  many 
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deficiencies  were  detected  late  in  the  pro¬ 
curement  process  that  an  independent 
Flight  Test  Division  was  established  to  con¬ 
duct  test  and  evaluation  independent  of  the 
contractors  and  project  offices.  To  meet  the 
need  for  practitioners  of  this  independent 
testing,  the  military  established  a  test  pilot 
school  to  improve  technical  competencies 
and  standardize  flight-test  methodologies.2 

By  the  end  of  the  twentieth  century,  ad¬ 
vances  in  technology,  political  shifts  in  ac¬ 
quisition  policy  and  funding  levels,  and 
mission  requirements  had  affected  the  bal¬ 
ance  of  roles,  responsibilities,  and  authority 
between  government  and  industry  testers. 

A  series  of  acquisition-reform  initiatives  in 
the  1990s  generally  decreased  government 
involvement  in  test  planning,  execution, 
and  reporting.  At  best,  government  testers 
became  partners  in  the  conduct  and  analy¬ 
sis  of  tests.  At  worst,  they  simply  evaluated 
test  results  for  the  program  office,  resulting 
in  a  significant  reduction  of  experienced 
government  test  personnel  and  a  veritable 
freeze  in  accessing,  training,  and  educating 
the  next  generation  of  test  professionals.3 
"The  lack  of  skilled  oversight  is  costing  the 
government,”  notes  Sue  C.  Payton,  the  pre¬ 
vious  assistant  secretary  of  the  Air  Force  for 
acquisition.  "I  could  save  millions  of  tax¬ 
payer  dollars  .  .  .  but  I  have  to  have  the 
workforce  with  the  domain  knowledge  that 
could  be  able  to  oversee  it  and  manage  it."4 

Senators  Carl  Levin  (D-MI)  and  John 
McCain  (R-AZ)  of  the  Senate  Armed  Services 
Committee  introduced  the  Weapon  System 
Reform  Act  of  2009  in  order  to  "remedy  a 
fundamentally  broken  defense  acquisition 
system."5  The  defense  acquisition  program 
suffered  from  a  loss  of  resident  expertise  in 
the  1990s  and  a  lack  of  involvement  of  test 
professionals  early  in  the  process.  This,  along 
with  other  political,  fiscal,  and  technical 
factors,  has  resulted  in  a  series  of  major  ac¬ 
quisition  programs  that  cannot  be  executed 
either  on  budget  or  on  time,  thus  degrading 
the  ability  of  the  war  fighter  to  respond  rap¬ 
idly  to  emerging  threats  and  maintain  supe¬ 
riority  in  a  turbulent  world.  "I  can't  tell  you 
how  many  programs  have  come  to  me  that 


aren't  signable  because  they  are  improperly 
structured  or  funded,”  says  John  J.  Young,  the 
previous  deputy  undersecretary  of  defense 
for  acquisition,  technology,  and  logistics.6 

The  Air  Force’s  acquisition  workforce  de¬ 
clined  from  57,000  personnel  20  years  ago 
to  24,000  at  the  end  of  2008. 7  According  to 
Payton,  "If  you  look  at  the  workforce,  we 
were  up  around  500,000  people  in  acquisi¬ 
tion  in  all  of  the  Defense  Department.  It  is 
down  to  about  200,000  now.  .  .  .  What  we 
are  managing  is  scarcity.”8  This  scarcity  re¬ 
fers  not  only  to  the  total  workforce  but  also 
to  the  proportion  of  government  testers, 
which  has  declined  compared  to  contractor 
personnel.  The  latter  comprised  20  percent 
of  the  acquisition  workforce  in  1994,  a  ratio 
that  more  than  doubled  to  50  percent  in 
2003,  thereby  creating  a  dependence  of  in¬ 
experienced  government  officials  on  con¬ 
tractors.  In  the  last  15  years,  many  programs 
have  been  adversely  affected  by  poor  judg¬ 
ment  that  can  be  attributed  to  an  inexperi¬ 
enced  acquisition/test  workforce  and  fund¬ 
ing  reductions.9  The  Air  Force  is  not  alone 
in  its  predicament;  all  of  the  services  pro¬ 
duced  underfunded  programs,  offered 
poorly  built  budgets,  and  underestimated 
requirements  as  preludes  to  seeking  a  cash 
infusion  from  the  Office  of  the  Secretary  of 
Defense.10  The  problems  seen  in  the  de¬ 
fense  acquisition  corps  in  general  are  also 
felt  in  the  developmental  test  and  evalua¬ 
tion  enterprise: 

•  A  large  number  of  the  most  experi¬ 
enced  management  and  technical  per¬ 
sonnel  in  government  and  industry 
were  lost  with  no  adequate  replace¬ 
ment  pipeline.11 

•  Major  personnel  reductions  strain  the 
pool  of  the  government's  experienced 
test  personnel.  A  significant  amount  of 
developmental  testing  occurs  without 
an  appropriate  degree  of  government 
involvement  or  oversight  and,  in  some 
cases,  with  limited  government  access 
to  contractor  data.12 
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•  The  number  of  Air  Force  test  personnel 
has  declined  by  approximately  15  per¬ 
cent,  and  engineering  personnel  in 
supporting  program  offices  have  been 
reduced  by  as  much  as  60  percent  in 
some  organizations.  Moreover,  these 
reductions  occurred  during  a  time 
when  programs  have  become  increas¬ 
ingly  complex.13 

The  Benefits  of 
Test  Professionals 

Tbst  professionals  must  appreciate  their 
often  unrecognized  leadership  roles  and 
carefully  apply  their  substantial  respon¬ 
sibilities. 

— Lt  Col  E.  John  Tfeichert 
“Testing  Efficacy:  The 
Substantial  Influence  of 
Test  Professionals” 

Upon  taking  office  as  the  19th  chief  of 
staff  of  the  Air  Force  in  August  2008,  Gen 
Norton  A.  Schwartz  identified  acquisition 
excellence  as  one  of  his  top  initiatives.14  A 
critical  part  of  any  proposed  solution  to 
General  Schwartz's  challenge  is  the  deliber¬ 
ate  development  of  a  cadre  of  test  profes¬ 
sionals.  As  a  subset  of  the  larger  defense 
acquisition  corps,  these  professionals  de¬ 
liver  capabilities  and  value  critical  to  an  ef¬ 
fective  acquisition  program.  The  skills  of 
the  test  professional  must  be  applied  across 
the  acquisition  process,  from  the  initial  gen¬ 
eration  of  requirements  to  the  sustainment 
of  weapons  systems. 

Tfest  professionals'  dedication  to  the 
needs  of  the  war  fighter  is  critical  to  their 
ability  to  translate  needed  war-fighting  ca¬ 
pabilities  into  a  set  of  requirements.  These 
needs  serve  as  the  genesis  of  a  reliable  sys¬ 
tem  that  functions  effectively  and  effi¬ 
ciently  in  the  intended  operational  environ¬ 
ments  against  known  and  conceivable 
threats.  The  test  professional's  early  in¬ 
volvement  in  the  acquisition  process  can 
help  focus  research  efforts,  define  test  as¬ 
sets,  assess  technical  risks,  determine  test 


resources,  and  scope  the  test  program.  It  is 
critical  that  such  professionals  become  in¬ 
volved  in  the  generation  of  requirements 
before  the  Joint  Requirements  Oversight 
Council  locks  them  in.  Several  acquisition 
programs  (e.g.,  the  Joint  Air-to-Sur face 
Standoff  Missile  and  the  Space-Based  Infra¬ 
red  System)  significantly  exceeded  their 
budgets  partly  due  to  poorly  written,  unre¬ 
alistic  requirements.15  Test  professionals  are 
particularly  suited  to  aligning  operational 
requirements  with  test-related  evaluations 
that  verify  and  validate  a  system  design. 
That  process  is  often  heuristically  based 
and  heavily  influenced  by  their  military 
judgment  and  prior  test  experience. 

The  current  trend  in  industry  to  protest 
source-selection  decisions  serves  as  an 
added  impetus  for  developing  well-defined, 
verifiable  requirements.  Poorly  articulated 
metrics  have  contributed  to  embarrassing 
bid  protests,  such  as  the  $35  billion  Air 
Force  KC-X  tanker-replacement  debacle.16 
Such  protests  are  "dragging  us  down  to  the 
nth  degree,"  Payton  observes.  "Acquisition 
folks  have  not  taken  adequate  measures  to 
make  sure  requirements  are  testable  and 
verifiable  in  contract  award.”17  The  acquisi¬ 
tion  community  and  test  professionals  are 
now  held  to  the  practical  standard  of  writing 
requirements  that  are  of  practical  use  by  a 
source-selection  authority  and  unassailable 
in  court.  Anything  less  will  cause  delays  of 
needed  capability  to  the  war  fighter. 

Jfest  and  evaluation  is  perfectly  situated 
to  significantly  affect  the  life-cycle  cost  of  a 
system— at  the  crossover  of  cost  and  risk 
(see  figure).18  The  economies  of  detecting 
design  deficiencies  and  implementing  solu¬ 
tions  on  only  a  handful  of  test  articles,  com¬ 
pared  to  implementing  a  solution  on  a 
fielded  system,  support  the  cost  of  main¬ 
taining  a  developmental  test  capability. 
Roughly  75  percent  of  a  system's  life-cycle 
costs  are  set  in  the  initial  design  process,  so 
an  early,  rigorous  test  program  will  save 
time  and  money  over  the  life  cycle  of  the 
system.19  In  both  the  development  of  a  new 
system  and  the  long-term  sustainment  efforts 
that  follow,  test  professionals  are  critical  to 
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Laboratory  Flightiest  Operations 


Wind  Tunnel  Production 

Figure.  Test  at  the  crossover  of  cost  and  risk.  (Reprinted  from  Aaron  A.  Tucker  and  Cihan  H.  Dagli,  "Design 
of  Experiments  as  a  Means  of  Lean  Value  Delivery  to  the  Flight  Test  Enterprise,"  Journal  of  Systems  Engineer¬ 
ing  12,  no.  3  [forthcoming],  203.) 


ensuring  that  the  system  is  fully  and  accu¬ 
rately  tested  and  evaluated.  Payton  observes 
that  "it's  more  beneficial  in  the  long  run  to 
spend  an  additional  20  percent  on  a  pro¬ 
gram  in  the  development  phase  (including 
prototypes  or  flyoffs)  than  to  pay  for  58  per¬ 
cent  overruns  in  the  future  when  a  project 
is  found  to  be  lacking  in  technology  or  test 
procedures."20  As  test  articles  are  designed 
and  built,  programmatic  risk  begins  to  de¬ 
crease  because  design  choices  have  been 
bounded  or  selected,  technology  has  ma¬ 
tured,  and  cost  and  schedule  uncertainties 
come  into  focus. 

Introducing  the  Weapon  Acquisition  Re¬ 
form  Act  of  2009,  Senator  McCain  noted 
that  the  "key  to  defense  acquisition  pro¬ 
grams  performing  successfully  is  getting 
things  right  from  the  start— with  sound  sys¬ 
tems  engineering,  cost-estimating,  and  de¬ 
velopmental  testing  early  in  the  program 
cycle.”21  Integration  of  test  professionals  at 
the  earliest  stages  of  requirements  generation 
is  essential  in  order  to  realize  the  benefits 


of  systems  engineering  by  tracing  measur¬ 
able  requirements  through  test  to  delivery 
of  the  capability.  The  skills  that  such  indi¬ 
viduals  bring  to  the  development  team  aug¬ 
ment  and  focus  the  program  manager's  task 
of  managing  the  cost,  schedule,  and  perfor¬ 
mance  of  a  system.  Tightly  controlled  per¬ 
formance  metrics  help  rein  in  cost  and 
schedule  expansion.  Excluding  test  person¬ 
nel  and  their  experience  from  the  develop¬ 
ment  phase  is  a  short-sighted  attempt  to 
save  money  and  results  in  increased  life- 
cycle  costs.22  While  war  fighters  operate 
their  equipment  as  established  systems  re¬ 
plete  with  the  inertia  that  makes  change 
difficult,  test  professionals  can  affect  a  sys¬ 
tem  design  when  changes  are  still  relatively 
cheap  and  easy.23  Further,  each  system 
must  be  considered  as  part  of  the  larger, 
networked  battlespace  and  integrated  into  a 
system  of  systems,  which  is  most  easily  ac¬ 
complished  early  in  the  process.  Modern 
systems  of  systems  fuse  information  from 
sensors  across  the  battlespace,  from  ground 
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to  air  to  space.  Fully  testing  such  a  capability 
substantially  increases  the  complexity  and 
expense  of  the  test  with  each  added  sensor, 
which  gives  further  impetus  for  the  early 
involvement  of  test  professionals. 

Just  as  air  systems  demand  thorough 
testing  to  ensure  their  safe  and  effective  op¬ 
eration,  space  equipment  also  requires  rigor¬ 
ous  testing  which  is  encumbered  by  pecu¬ 
liar  challenges.  Space  systems  in  orbit  are 
unique  pieces  of  hardware  that  are  subject 
to  a  particularly  unforgiving  environment 
and  generally  cannot  be  directly  accessed 
once  placed  into  service.  These  systems  are 
exposed  to  thermal  shock  and  atmospheric 
extremes  that  are  difficult,  if  not  impossible, 
to  test  accurately  before  launch.  Few,  if  any, 
identical  systems  are  produced,  and  no 
ability  exists  to  correct  discrepancies  dis¬ 
covered  after  launch.  Thermal/ vacuum 
testing,  one  of  the  final  evaluations  of  or¬ 
bital  systems,  offers  the  best  approximation 
of  the  hard  vacuum  of  space.  Such  fidelity, 
however,  remains  extremely  expensive  and 
takes  weeks  to  execute  in  one  of  a  handful 
of  facilities  in  the  country.  The  availability 
of  thermal/vacuum  chambers  that  can  ac¬ 
commodate  large  satellites  is  particularly 
limited.  Integration  testing  of  the  orbital 
system  and  ground  control  is  also  very  im¬ 
portant.  These  system-level  tests  account 
for  35  to  50  percent  of  nonrecurring  costs.24 
Tfest  professionals  with  operational  experi¬ 
ence  are  particularly  critical  in  space  acqui¬ 
sition  programs  because  they  occupy  the 
best  position  for  discovering  discrepancies 
and  correcting  them  before  a  system  is 
placed  in  orbit. 

Software  is  one  of  the  few  systems  that 
can  be  developed  and  maintained  after  the 
launch  of  a  space  system.  In  the  last  two 
decades,  systems  have  become  increasingly 
software  intensive.  In  order  to  manage  the 
complexity  of  software-intensive  systems, 
many  programs  have  adopted  a  block-up¬ 
grade  strategy  whereby  each  upgrade  drives 
its  own  developmental  test  program,  which 
merges  into  almost  continuous  test  programs 
(e.g.,  F-16  Block  Upgrades,  C-17  Follow-on 
Flight  Test  Program,  and  Global  Positioning 


System  Blocks  I  through  III).  Sustainment 
test  programs  maintain  a  system's  rele¬ 
vancy  and  require  the  continuous  involve¬ 
ment  of  test  professionals  with  a  steady  fo¬ 
cus  on  requirements  and  test  discipline. 
These  personnel  must  ensure  delivery  of 
the  new  capability  in  a  block  upgrade  and 
prevent  the  degradation  of  baseline  capa¬ 
bilities  through  regression  testing. 

The  value  of  test  professionals  corre¬ 
sponds  to  systems-engineering  principles 
which  hold  that  programmatic  risk  and  un¬ 
certainty  are  probabilities  that  can  be  miti¬ 
gated  or  eliminated.  Their  value  lies  in  the 
independent  evaluation  of  system  perfor¬ 
mance,  which  supports  fielding  decisions. 
Tfest  professionals  help  generate  require¬ 
ments,  evaluate  acquisition  proposals,  and 
offer  their  expert  insight  into  technology 
and  performance  risk  rather  than  simply 
select  the  lowest-cost  proposals.  If  these  in¬ 
dividuals  fail  to  perform  their  duties  prop¬ 
erly,  the  needed  change  may  prove  techni¬ 
cally  impossible  or  fiscally  prohibitive.25 
Similarly,  the  time  for  the  system’s  effec¬ 
tiveness  may  have  passed,  resulting  in  a 
defeat  on  the  battlefield,  the  fielding  of  an 
enemy  countermeasure,  or  a  paralyzing 
war  of  attrition.  Jfest  professionals  with  an 
operational  focus  can  break  through  crip¬ 
pling  limitations  by  questioning  assump¬ 
tions  and  applying  technology  to  provide 
new  capabilities. 

Efficient  programmatic  practices  are  in 
continual  demand  from  the  test  profes¬ 
sional:  risk  management,  test  planning, 
mission  relevance,  deficiency  reporting, 
and  programmatic  wherewithal.26  Test  pro¬ 
fessionals  must  develop  an  ability  to  under¬ 
stand  and  balance  cost,  schedule,  perfor¬ 
mance,  and  their  attendant  risks  and 
uncertainties.  An  understanding  of  the 
needs  of  the  war  fighter  is  critical  to  deci¬ 
sions  about  performance  risk.  Which  capa¬ 
bilities  can  be  cancelled,  delayed,  or  modi¬ 
fied,  and  which  are  not  negotiable?  Test 
professionals  have  a  unique  perspective 
that  allows  them  to  find  problems  or  defi¬ 
ciencies  before  a  fielding  decision  is  made, 
to  evaluate  design  fixes,  and  to  prevent  re- 
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work  on  production  systems.  Even  within  a 
single  developmental  test  program,  a 
skilled,  experienced  test  team  can  save  time 
and  money  by  reducing  the  fly-fix-fly  cycle. 
Developmental  test  is  expensive  but  not 
nearly  as  costly  as  not  having  skilled,  expe¬ 
rienced  test  professionals.  The  price  of  find¬ 
ing  a  deficiency  late  in  a  system’s  life  cycle 
and  then  implementing  a  design  change 
can  be  quite  high.27  For  instance,  space  sys¬ 
tem  programs  spent  10  percent  of  the  devel¬ 
opment  schedule  and  10  percent  of  their 
profit  margins  fixing  problems  not  discov¬ 
ered  until  the  final  system-level  thermal/ 
vacuum  test.28 

Maj  Gen  David  J.  Eichhorn,  commander 
of  the  Air  Force  Flight  Test  Center,  believes 
that  the  "government's  role  can't  be  allowed 
to  degrade  into  nothing  more  than  deep 
pockets  /check  writers."29  Complete  infor¬ 
mation  informs  the  decisions  of  acquisition 
authorities  as  they  continually  balance  cost, 
schedule,  and  performance  while  steering  a 
direct  course  to  deliver  combat  capability  to 
the  war  fighter.  Tfest  professionals  have  the 
responsibility  of  collecting  and  interpreting 
rigorous  technical  data  from  the  earliest 
analyses  of  materiel  solutions  and  technology- 
development  efforts  through  sustainment. 
They  should  then  educate  acquisition  deci¬ 
sion  makers  on  the  underlying  assumptions 
and  probabilities  associated  with  the  sys¬ 
tem.  Even  before  actual  test  data  is  avail¬ 
able  for  a  system,  test  professionals  can  ad¬ 
vise  decision  makers  using  judgment  born 
of  education,  training,  and  experience  as 
practical  testers.  Source-selection  teams  can 
leverage  the  judgment  of  these  professionals 
to  evaluate  proposed  test  programs.30 

Tbols  such  as  design  of  experiments  (DOE) 
and  theory  of  constraints  have  been  applied 
to  overcome  the  debilitating  need  for  abso¬ 
lute  surety  and  the  distractions  of  false  di¬ 
lemmas.  Both  tools  employ  a  statistically 
rigorous  analysis  to  determine  the  probability 
that  a  particular  reality  actually  exists,  based 
on  a  finite  number  of  observations.  DOE- 
based  test  plans  enable  the  development  of 
analyses  and  conclusions  couched  in  terms 
of  statistical  confidence  and  power  intervals. 


These  statistical  measures  of  the  quality  of 
test  data  are  critical  to  sound,  objective  ac¬ 
quisition  decisions.  Further,  test  professionals 
can  present  decision  makers  with  discrete 
levels  of  test  resources  required  to  answer  a 
particular  question— essentially  buying  in¬ 
crements  of  statistical  confidence  and 
power.31  One  case  study  proposes  that  a 
DOE-based  flight-test  experiment  can  save 
70  to  84  percent  of  the  cost  of  traditional, 
one-factor-at-a-time  approaches.32 

Balance  within  a 
Cadre  of  Test  Professionals 

Scientific  results  cannot  be  used  efficient¬ 
ly  by  soldiers  who  have  no  understand¬ 
ing  of  them,  and  scientists  cannot  pro¬ 
duce  results  useful  for  warfare  without 
an  understanding  of  the  operations. 

—Dr.  Theodore  von  Karman 
Toward  New  Horizons 

Tfest  professionals,  who  have  a  variety  of 
technical  skill  sets,  include  operators,  engi¬ 
neers,  and  program  managers  trained  and 
educated  in  the  art  and  science  of  test.  Each 
career  path  should  be  developed  within  a 
cadre  of  test  professionals  comprised  of  a 
balance  of  two  types  of  experts: 

1.  System/mission  experts  who  have 
depth,  recency,  and  career  focus  in 
operations  coupled  with  firsthand  test 
experience. 

2.  Developmental  test  experts  who  may 
have  a  background  in  operations  and 
maintain  a  career  path  focused  on  de¬ 
velopmental  test. 

Both  types  of  experts  are  operator,  engi¬ 
neer,  and  program-manager  members  of  a 
combined  test  force  (CTF),  which  can  focus 
on  a  system  (e.g.,  an  F-35  CTF)  or  a  capability 
(e.g.,  a  Global  Reach  CTF  or  a  Global  Power 
CTF).  All  members  of  a  CTF  contribute  to 
the  developmental  test  and  evaluation  pro¬ 
gram  to  develop  capabilities  for  the  war 
fighter.  System/mission  experts  provide  ex¬ 
tensive  system  expertise  to  evaluate  new 
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capabilities  and  support  the  CTF's  training, 
standardization,  and  operations  functions. 
Developmental  test  experts  act  to  ensure 
that  systems  are  evaluated  safely,  effec¬ 
tively,  and  efficiently  through  test  and 
safety  planning  and  reporting.  Both  share  in 
the  execution  of  test  missions  according  to 
their  specific  skill  sets— by  exchanging  ideas 
and  experience,  they  enhance  the  CTF  mis¬ 
sion  of  providing  decision-quality  data  for 
acquisition  programs. 

System/mission  experts  should  be 
closely  identified  with  the  operational  com¬ 
munity.  The  Defense  Science  Board's  report 
on  developmental  test  and  evaluation  rec¬ 
ommends,  as  a  minimum,  making  available 
a  cadre  of  operational  personnel  to  support 
developmental  test  and  evaluation  for  Ac¬ 
quisition  Category  I  (total  procurement  of 
more  than  $2.19  billion)  and  special-interest 
programs.33  System/mission  experts  can  en¬ 
sure  that  evaluations  are  conducted  in  the 
context  of  the  mission,  which  can  be  evolv¬ 
ing  with  emerging  threats  and  new  tactics, 
techniques,  and  procedures.  They  would 
evaluate  the  system  in  terms  of  mission  ca¬ 
pability  and  report  the  results  in  terms  of 
operational  significance  to  the  user.34  This 
cadre  brings  operational  considerations 
such  as  the  utility  of  new  capabilities  to  the 
developmental  test  program  and  seeds  the 
future  ranks  of  senior  leaders  with  officers 
who  have  working-level  experience  in  test 
and  acquisition.  A  National  Research  Council 
study  of  2008  characterizes  inexperienced 
government  and  industry  personnel  in  key 
leadership  positions  as  the  largest  driver  of 
cost-development  time  and  performance 
risk.35  A  continuous  flow  of  recent  opera¬ 
tional  expertise  to  the  test  enterprise  is  jus¬ 
tified  by  considering  the  benefits  to  the  ac¬ 
quisition  programs  and  the  professional 
development  of  the  individuals. 

System/mission  experts'  professional  de¬ 
velopment  broadens  from  a  concentration 
on  operations  to  include  an  acquisition  per¬ 
spective.  After  one  or  two  operational  as¬ 
signments,  an  operator  with  a  technical 
background  and  experience  as  an  instructor 
in  a  major  weapons  system  is  eligible  to  join 


the  cadre  of  test  professionals.  For  indi¬ 
viduals  with  solid  operational  credentials, 
an  assignment  in  test  and  evaluation  could 
become  an  alternative  to  a  tour  as  a  school- 
house  instructor  or  air  liaison  officer.  Weap¬ 
ons  school  graduates  would  be  particularly 
valuable  to  a  test  organization.  The  Defense 
Acquisition  University’s  online  courses  in 
acquisition  and  test  and  evaluation  would 
serve  as  an  entree  for  novice  test  profes¬ 
sionals,  and  training  in  a  National  Test  Pilot 
School  or  an  Air  Force  Test  Pilot  School 
short  course  could  train  operators  and  flight 
test  engineers  for  operational  and  develop¬ 
mental  test  assignments.  Moreover,  flight- 
test  assignments  that  perform  program 
management  could  provide  staff  officer  ex¬ 
perience  for  senior  captains  or  junior  majors, 
coupled  with  flying  duties.  Although  out  of 
the  air  and  space  expeditionary  force's  de¬ 
ployment  cycle  for  their  weapons  systems, 
test  professionals  could  support  individual 
deployment  taskings  commensurate  with 
their  skill  sets,  enabling  them  to  stay  con¬ 
nected  with  current  operations  and  shoulder 
their  fair  share  of  the  deployed  mission. 

The  breadth  of  acquisition  experience 
gained  by  a  system/mission  expert  depends 
largely  on  the  program,  but  most  test  pro¬ 
fessionals  would  become  familiar  with  and 
have  the  opportunity  to  affect  several  pro¬ 
grams  in  different  stages  of  the  acquisition 
process  before  returning  to  the  war-fighting 
commands.  Along  with  taking  Defense  Ac¬ 
quisition  University  courses,  this  experience 
would  qualify  the  individual  for  an  Acquisi¬ 
tion  Professional  Development  Program 
Level  II  or  III  certification  in  test  and  evalua¬ 
tion.36  The  courses  and  training  that  lead  to 
these  certifications  would  help  system/mis¬ 
sion  experts  understand  the  capabilities  and 
limitations  of  operational  and  developmental 
test  and  evaluation.  Additionally,  acquisition 
certifications  and  test  experience  would  ex¬ 
pand  their  eligibility  for  higher-level  staff 
assignments  in  test,  acquisition,  plans,  pro¬ 
grams,  and  operational  tactics  and  training. 
Finally,  due  to  their  involvement  with  next- 
generation  systems,  these  experts  would 
become  very  familiar  with  the  newest  sys- 
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tem  capabilities  and  would  be  uniquely 
qualified  to  deliver  a  system  to  the  war¬ 
fighting  command  as  the  initial  cadre  in  a 
leadership  capacity.  These  rising  leaders 
would  be  able  to  draw  on  their  direct  expe¬ 
rience  with  acquisition  as  they  progress  to 
roles  of  increasing  responsibility.  The  Air 
Force  should  emphasize  the  value  of  a  test 
and  evaluation  tour  to  ensure  that  system/ 
mission  experts  are  promoted  to  augment 
the  ranks  of  senior  leaders  with  individuals 
who  are  able  to  draw  on  their  direct  experi¬ 
ence  with  acquisition  as  they  progress  to 
roles  of  increasing  responsibility. 

Acquisition  programs  benefit  from  the 
valuable,  recent  operational  experience  of 
system/mission  experts.  Furthermore, 
these  personnel  can  be  drawn  from  the  gen¬ 
eral  pool  of  operators,  engineers,  and  pro¬ 
gram  managers,  thus  providing  a  flexible, 
responsive  manning  source  from  which  to 
quickly  increase  or  decrease  the  manning 
according  to  the  needs  of  the  particular  test 
program.  The  inclusion  of  system/mission 
experts  in  a  cadre  of  test  professionals  also 
greatly  enhances  the  amount  of  operational 
expertise  organic  to  the  acquisition  program. 
Finally,  system/mission  experts  who  are 
operators  can  participate  in  the  vast  majority 
of  test  missions  because  only  medium-  and 
high-risk  test  missions  (12  percent  of  test 
sorties)  require  graduates  of  a  test  pilot 
school  to  execute  the  mission.37  The  fact 
that  that  requirement  may  be  met  by  con¬ 
tractors  or  waived  by  the  test  leadership  fur¬ 
ther  increases  the  opportunity  for  system/ 
mission  experts  to  execute  test  missions.38 

Drawing  on  their  extensive  knowledge  of 
systems  and  tactics  in  major  weapons  sys¬ 
tems,  operator  system/mission  experts  can 
serve  as  instructors  or  evaluators  for  the  CTF 
and  as  command  chief  pilots  for  Air  Force 
Materiel  Command.  They  must  take  care, 
however,  to  overcome  the  philosophy  of  rigid 
training  and  standardization  rules  necessary 
in  operational  units.  The  developmental- 
test  mission  demands  flexibility  in  order  to 
execute  tests  safely  and  efficiently.  This 
flexibility  is  enabled  by  test  discipline,  tech¬ 
nical  judgment,  and  outstanding  airman¬ 


ship  of  highly  experienced  aircrews.  Tfest  is 
not  executed  by  inexperienced  copilots  or 
basic  wingmen.  The  learning  curve  is  al¬ 
ways  very  steep,  test  professionals  are 
rarely  comfortable,  and  each  person  must 
carefully  manage  operational  risk  as  it  re¬ 
lates  to  the  specific  test  mission.  The  risk  of 
realizing  a  hazard  is  also  carefully  mitigated 
by  the  operating  environment  (e.g.,  day¬ 
time,  good  weather,  sanitized  airspace,  and 
very  long  runways),  a  mission  profile  that 
has  been  vetted  through  multiple  levels  of 
technical  and  safety  reviews,  and  the  di¬ 
verse  team  of  experts  charged  with  plan¬ 
ning,  executing,  and  monitoring  highly  in¬ 
strumented  test  vehicles. 

System/mission  experts  complement  de¬ 
velopmental  test  experts  within  a  CTF.  The 
system/mission  expert's  career  is  weighted 
heavily  toward  operational  assignments, 
whereas  the  developmental  test  expert 
starts  with  a  technical  background,  adds  op¬ 
erational  experience,  and  continually  builds 
momentum  with  assignments  in  test  and 
acquisition  in  order  to  mature  as  an  acquisi¬ 
tion  professional.  The  developmental  work¬ 
force  tends  to  be  relatively  static  due  to  the 
extremely  long  lead  time  needed  to  select 
and  train  developmental  test  experts.  To  be 
effective,  they  should  start  developmental 
test  assignments  early  in  their  careers  after 
beginning  with  a  base  of  operational  exper¬ 
tise  upon  which  to  develop  skills  and  expe¬ 
rience.  Operators,  engineers,  and  program 
managers  who  are  growing  as  develop¬ 
mental  test  experts  need  to  learn  their  craft 
through  a  combination  of  education,  train¬ 
ing,  and  experience  while  undertaking  a 
series  of  increasingly  difficult  tasks.  Their 
professional  development  includes  honing 
critical-thinking  skills,  technical  acumen, 
and  engineering  judgment.  The  challenge 
involves  developing  their  ability  to  move 
flexibly  among  developmental  test  pro¬ 
grams  and  provide  effective,  system-generic 
test  expertise  while  remaining  operationally 
relevant.  Balanced  experience  across  major 
weapons  systems  is  a  critical  skill  for  devel¬ 
opmental  test  experts  to  possess. 
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The  value  of  the  dedicated  test  profes¬ 
sional  becomes  evident  when  designing  or 
executing  a  critical  test  point.  A  system 
must  demonstrate  its  capabilities  near  the 
edge  of  the  operating  envelope  when  sig¬ 
nificant  resources  are  at  stake.  Examples 
include  a  maximum-performance  braking 
event  when  tire  and  wheel  damage  is  ex¬ 
pected,  maximum  weight  operations  on  a 
dirt  landing  zone,  or  the  release  of  an  ex¬ 
pensive  weapon  at  the  edge  of  the  operating 
envelope.  Graduates  of  test  pilot  school  are 
the  best  candidates  for  assessing  technical 
and  safety  risks  in  order  to  ensure  that  the 
test  is  designed  and  executed  properly  the 
first  time.  Their  training  allows  them  to  de¬ 
sign  the  test  based  on  theory  enabled  by  a 
sense  of  what’s  actually  practical.  When 
executing  the  test,  operator  and  engineer 
developmental  test  experts  approach  the 


the  simple  goal  of  training  a  skill  set  by  also 
educating  a  test  professional's  critical  think¬ 
ing  and  judgment.  For  example,  the  US  Air 
Force  Test  Pilot  School's  curriculum  re¬ 
ceived  approval  to  begin  granting  a  master 
of  science  degree  in  flight-test  engineering, 
starting  in  May  2008.  Intermediate  Develop¬ 
mental  Education  in-residence  credit  as  well 
as  Defense  Acquisition  University  equiva¬ 
lency  (up  to  Level  III  Test  and  Evaluation 
coursework)  had  already  been  approved. 
This  trend  toward  strategic  education  sup¬ 
ports  the  progression  of  a  developmental 
test  expert.  Test  pilot  school  selection  boards 
consider  demonstrated  officership  as  well 
as  strong  academic  performance  in  the  ap¬ 
plied  sciences.  They  don't  simply  select  a 
test  pilot  school  student  but  a  future  devel¬ 
opmental  test  professional.  Tfest  professionals 
progress  to  command  test  and  development 


The  years  of  technical  development  and  training 
in  the  test  skill  set  produce  a  developmental  test 
expert  who  makes  decisions  and  gathers  data 
that  is  well  worth  the  cost  of  training. 


test  point  with  a  situational  awareness  de¬ 
veloped  toward  controlling  dynamic,  multi¬ 
variate  systems.  This  enables  them  to  ob¬ 
serve  the  test  as  well  as  overall  system 
performance  and  report  on  the  test  with  the 
benefit  of  years  of  trained  observation.  De¬ 
velopmental  test  experts  can  meet  the  chal¬ 
lenge  of  maintaining  operational  relevance 
by  reserving  time  for  participation  in  major 
exercises  or  operational  deployments. 

The  common  thread  among  the  syllabi  at 
all  test  pilot  schools  is  that  theoretical  ex¬ 
pertise  supports  safe,  effective,  and  efficient 
flight  test  and  accurate  reporting.  Each 
school  strikes  its  own  balance  of  instruction 
in  performance,  handling  qualities,  and  sys¬ 
tems.  They  all,  however,  attempt  to  surpass 


centers;  hold  senior  acquisition,  planning, 
and  programming  positions;  or  step  into  re¬ 
search  to  provide  operational  and  test  per¬ 
spectives  to  technology-development  ef¬ 
forts.  In  addition,  the  military  test  pilot 
schools  are  considered  strategic  assets  be¬ 
cause  they  provide  a  flow  of  expertise  into 
industry  as  well  as  into  the  government  test 
establishment.39 

The  years  of  technical  development  and 
training  in  the  test  skill  set  produce  a  de¬ 
velopmental  test  expert  who  makes  deci¬ 
sions  and  gathers  data  that  is  well  worth  the 
cost  of  training.  A  flight-test  engineer  can 
payback  those  training  costs  by  designing  a 
test  plan  that  safely  and  effectively  vali¬ 
dates  a  system's  capabilities.  A  test  pilot  can 
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justify  those  training  costs  by  executing  the 
test  point  on  the  first  attempt  and  by  accu¬ 
rately  reporting  the  results.  A  cadre  of  de¬ 
liberately  developed  test  professionals  justi¬ 
fies  its  cost  many  times  over  by  enabling 
acquisition  decisions  based  on  rigorous,  ac¬ 
curate  data  from  a  source  that  protects  the 
interests  of  the  war  fighter  and  taxpayer. 

Conclusion 

Better  be  prepared  to  dominate  the  skies 
above  the  surface  of  the  earth  or  be  pre¬ 
pared  to  be  buried  beneath  it. 

—Gen  Carl  A.  Spaatz 

The  chief  of  staff  of  the  Air  Force's  initia¬ 
tive  to  regain  acquisition  excellence  recog¬ 
nized  that  Congress  and  the  Department  of 
Defense  had  lost  confidence  in  the  service's 
acquisition  decisions  at  a  time  when  re¬ 
sources  must  be  carefully  conserved.  Tfest 
professionals  are  critical  to  providing  accu¬ 
rate  information  for  those  acquisition  deci¬ 
sions.  They  perform  the  necessary  function 
of  translating  needed  capabilities  to  require¬ 
ments,  managing  development  programs, 
and  accurately  and  fully  testing  systems. 
The  value  of  test  professionals  is  realized 
through  independent  evaluation  that  ex¬ 
poses  system  flaws  early  in  development 
when  they  can  be  solved  easily  and  quickly. 
They  also  produce  decision-quality  data  for 
acquisition  decision  makers  who  must  be 
able  to  rely  on  those  data.  Therefore,  it  is 
critical  that  a  cadre  of  deliberately  devel¬ 
oped  professional  testers  be  fully  integrated 
into  acquisition  from  the  earliest  stages. 


This  cadre  of  test  professionals  includes 
a  necessary  balance  of  system/mission  ex¬ 
perts  and  developmental  test  experts.  The 
former  include  operators,  engineers,  and 
program  managers  who  come  from  opera¬ 
tional  assignments  and  contribute  mission 
focus  and  system  expertise  to  test  programs 
before  returning  to  operational  assignments. 
They  can  gain  acquisition  experience  that 
will  prove  critical  later  in  their  careers  as 
senior  leaders  in  operations,  acquisition, 
plans,  or  programs.  Developmental  test  ex¬ 
perts  develop  core  skills  in  operations,  engi¬ 
neering,  and  program  management  that  are 
critical  to  planning  and  executing  safe,  ef¬ 
ficient,  and  effective  test  programs.  Their 
career  path  remains  in  test  and  acquisition 
to  take  advantage  of  experience  and  judg¬ 
ment  that  has  been  sharpened  by  the  chal¬ 
lenges  of  developmental  test. 

Fixing  the  problems  in  test  and  evaluation 
represents  a  complex  undertaking  yet  is  only 
a  small  part  of  achieving  acquisition  excel¬ 
lence.  Deliberate  development  and  invest¬ 
ment  in  the  acquisition  corps  in  general,  and 
in  the  test  professional  in  particular,  are 
necessary  for  the  Air  Force  to  answer  the 
chief  of  staffs  call.  Acquisition  excellence  is 
based  on  properly  navigating  a  series  of 
programmatic  decisions  fraught  with  risks 
and  assumptions.  Tfest  professionals  reduce 
those  risks  and  assumptions  with  data  and 
educate  the  judgment  of  decision  makers  to 
deliver  needed  capability  to  the  war  fighter 
and  secure  the  national  defense.  © 
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